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Abstract. While collaborative Intelligent Tutoring Systems (ITSs) have been 
designed for older students and have been shown to support sense-making be¬ 
haviors, there has not been as much work on creating systems to support collab¬ 
oration between elementary school students. We have developed and tested, 
with 84 students, individual and collaborative versions of an ITS that supports 
students’ learning of equivalent fractions. The current paper presents prelimi¬ 
nary data analysis focused on understanding what design patterns are associated 
with sense-making behaviors. Triangulating between log data, eye-tracking da¬ 
ta, and transcript data, we find evidence of sense-making behaviors for students 
who work collaboratively on procedural problems as shown by a correlation of 
increased levels of joint visual attention and conceptual learning gains, longer 
time spent on hints compared to students working individually, and example di¬ 
alogue. Future work will examine collaborative episodes more closely to isolate 
design features that are beneficial for elementary school students. 
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1 Introduction 

Collaborative learning has been shown to be beneficial in face-to-face and Computer 
Supported Collaborative Learning (CSCL) settings [6], [14]. However, supporting 
good collaboration, even in a CSCL environment, is challenging. Collaboration often 
does not happen spontaneously, and collaboration scripts are often used to support 
students in their learning by scaffolding the tasks and interactions between group 
members [3]. It is important for these scripts to match both the learning goals of the 
activity and the developmental level of the students. To date, much of the research on 
supporting collaboration has been done with older students and it is an open question 
how elementary school students can successfully take advantage of collaboration. To 
address this issue, we focus on how to best scaffold sense-making behaviors, as col¬ 
laboration is often successful when these learning behaviors are elicited. That is, 
when good collaboration occurs, we would expect to see more evidence of these be¬ 
haviors in dialogue, as well as in other data streams. By analyzing different data 



streams from elementary school student’s collaborations, we can better understand 
what differentiates successful dyads from those that are less successful and how best 
to support these interactions in a CSCL environment for this age group. 

Although the Common Core educational standards that most US states have adopt¬ 
ed expect elementary school students to collaborate, it is still unclear how to best 
provide collaborative support for this age group. Collaboration is used less in elemen¬ 
tary school classrooms, partially because of the additional challenge it can bring in 
terms of class management. Elementary school students are at a different develop¬ 
mental stage both cognitively and socially than older students, so it is unclear if the 
same types of support that are beneficial to older students would apply to younger 
students as well. There have been some successful collaborative paradigms that have 
focused on elementary school students, such as reciprocal teaching, but these are of¬ 
ten not supported in a CSCL environment [10]. 

We developed a collaborative Intelligent Tutoring System (ITS) to support the 
learning of fractions for 4 th and 5 th grade students. Support was provided for the 
collaboration through an embedded collaboration script designed to match the learn¬ 
ing goals of the problem sets. We found that students working collaboratively had 
equivalent learning gains to those who worked individually [9], To better understand 
how to improve the provided support for collaboration with elementary school stu¬ 
dents, we collected log data, eye-tracking data, and transcript data as students worked 
with our ITS. Because collaboration is often successful through students’ mutual 
elaborations and explanations that can lead to sense making, in the present paper, we 
examine this data for evidence of sense-making behaviors both within the transcripts 
and in terms of visual joint attention and time spent on a requested hint. By triangulat¬ 
ing this data and looking for the moments of good collaboration, we are better able to 
find what successful elementary school students do and how best to support these 
actions. We would expect to find more evidence of sense-making behaviors as stu¬ 
dents work on conceptually-oriented problems than procedurally-oriented problems 
because these behaviors would support the making of deeper connections needed for 
conceptual knowledge more than the fluency needed for procedural knowledge. 


2 Methods 

2.1 Collaboration Support 

Informed by prior work on the Fractions Tutor [11], we developed an ITS to support 
the learning of equivalent fractions. We developed two parallel versions, one for col¬ 
laborative learning and one for individual learning. Each version included the cogni¬ 
tive support that is typical of an ITS, such as step-by-step guidance and on-demand 
hints, while the collaborative tutor also provided support for collaboration through an 
embedded collaboration script. Each ITS included two problem sets targeted at the 
learning of conceptual knowledge or procedural knowledge. The collaborative tutors 
support synchronous collaboration, in which paired students sit at their own computer 
and have a shared view of the problem while collaborating through audio chat. 



The collaboration scripts consisted of a subset of three collaboration features: roles, 
unique information, and cognitive group awareness. Roles assigned specific tasks to 
group members while unique information provided different information to each stu¬ 
dent for a sense of accountability. Cognitive group awareness was supported through 
each student answering a question independently before seeing their partner’s answer 
and then choosing a group answer. The problems targeting conceptual knowledge 
included all three features, while procedural problems were supported through the 
collaboration features of roles and unique information. The different features were 
chosen for the conceptual and procedural problem sets to best support the type of 
knowledge being targeted. Cognitive group awareness was used to scaffold only the 
conceptual problems because providing the opportunity to discuss differing answers 
provides more support for behaviors that lead to sense making than to fluency. Be¬ 
cause there was more support for activities that can lead to sense making in the con¬ 
ceptual problems, we may expect to see more evidence of these behaviors in the con¬ 
ceptual problems than in the procedural problems. 

2.2 Experimental Design and Procedure 

Participants in the study were 84 4 th and 5 th grade students, ranging from 9-12 years 
old (M = 9.96, SD = .75) from two US elementary schools. The students participated 
in “pull-out” sessions (in a lab room set up in the school) during normal school hours 
where they worked for 45 minutes with a version of our ITS. Students were assigned 
to pairs based on their teachers’ pairings, and each pair was randomly assigned to one 
of four conditions, by crossing two factors; collaborative or individual instruction and 
problems geared towards conceptual knowledge or procedural knowledge. We 
developed two computer-based isomorphic test forms that matched the target 
knowledge taught in the tutors. 


3 Results 

To analyze the behaviors during the tutor task that may be indicative of sense making, 
we looked at the data logs, the eye-tracking data, and are beginning to analyze the 
transcripts from the collaborative groups. Across all data types, the procedural and 
conceptual conditions were analyzed separately because of the different tutor problem 
designs. The tutor log data was analyzed to interpret the hint behavior in the different 
conditions. Specifically, the time spent per hint was analyzed since hint requests often 
happen at times of problem solving impasses, which may be good occasions for learn¬ 
ing, and more time on hints can be a signal of behaviors that lead to sense making 
such as discussion or self-explanation. For the conceptual conditions, the students 
working individually (IC) requested significantly more hints than those working col- 
laboratively (CC), t(23.0) = 2.38, p = .03, while in the procedural conditions the stu¬ 
dents working individually (IP) asked for marginally more hints than the students 
working collaboratively (CP), t( 27.8) = 2.00, p = .06. However, for the two conceptu¬ 
al conditions, there was no significant difference between the CC and IC conditions 



on average time spent per hint requested, while for the procedural condition, there 
was a significant difference between the CP and IP conditions on the amount of time 
spent per hint, t( 26.7) = 2.64 , p = .01. The CP condition spent significantly more time 
per hint level with a mean of 13.38 seconds compared with the individual condition 
mean of 6.50 seconds (see Figure 1). This longer time spent on each hint may be a 
signal that more productive collaboration, which supports sense-making learning 
processes, was occurring during these collaborative procedural sessions. 
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Fig. 1 . Time spent per hint was significantly higher in the collaborative procedural condition. 

In addition to analyzing what tutor log data might reveal about student collabora¬ 
tion, we are exploring the utility of using joint visual attention measures as an index 
of collaboration quality, consistent with Niissli [8]. We extracted a measure of joint 
attention, defined as the amount of time the students spent looking at the same area of 
the screen at the same time. Because a variable number of problems were completed 
by each dyad, as a preliminary measure, we averaged the joint attention of the first 
four problems. Another paper provides a more detailed analysis of the eye-tracking 
data [1]. While joint attention correlated with total test scores (i.e., conceptual and 
procedural items), there was no correlation between the amount of joint attention and 
the procedural test items, r = .14,/? = .491. Surprisingly, for the procedural condition, 
there was a correlation between the joint attention and the conceptual test items, r = 
.35, p = .072, but this was not observed for the conceptual condition, r= .08 ,p =. 111. 
Since joint visual attention indicates that students are visually attending the same area, 
more joint attention may indicate that dyads are participating in activities that require 
both students’ attention, such as discussions that may lead to more sense making. 

We also have begun to analyze the transcripts from the audio data for moments of 
good collaboration as defined by past research [2], [7], [13]. To begin the preliminary 
analysis, we looked at an example dyad that had a high average duration per hint. 
Within this transcript, we saw signs of more interactive learning through the use of 
questions and answers. Below is an excerpt representative of some of the discussions 
between the students. Within context, the students have been asked to multiply 4/7 to 
find equivalent fractions. 







Student 2 :1 think it means the first fraction, um...times one...in the instructions 
Student 2: um... 

Student 1: 1...still don't...wait... 

Student 1: I still don't get it. 

Student 2: Um, try the purple fraction times the...times one 
Student 2 first explains to their partner in broader terms about how to complete the 
step and points out the information in the instructions before providing additional help 
with a more specific hint. 


4 Discussion and Conclusion 

Through preliminary data analysis, we have seen evidence of sense-making behaviors 
during collaborative procedurally-oriented tasks that may lead to better conceptual 
knowledge. During procedural problem solving, the students in the collaborative con¬ 
dition spent more time per hint than the students working individually. This difference 
on the duration spent on each hint might be because the students are spending more 
time discussing and making sense of the hint in the collaborative condition. In the 
eye-tracking data, we see that the students working collaboratively on procedurally- 
oriented problems, there is a correlation between joint attention and conceptual learn¬ 
ing gains. Since joint attention may be a measure of better collaboration and since we 
did not see this correlation in the conceptual instruction condition, collaboration may, 
surprisingly, have been more important to gain conceptual knowledge with procedur¬ 
ally-oriented problems than with conceptual-oriented problems. 

Conceptual and procedural learning are related through an iterative relationship 
[12]. Learning conceptual knowledge can transfer to procedural skills while learning 
procedural skills can transfer to conceptual knowledge. However, the relationship is 
not symmetrical and the gains may be greater when learning conceptual knowledge 
[12]. In our study, it may be that the procedural tutor provided less support for gaining 
conceptual knowledge leaving room for collaborating students to engage in produc¬ 
tive sense making, as illustrated in the example dialogue of students. Collaboration 
may be most successful when it supports activities that lead to more sense making, 
which can be beneficial for conceptual knowledge. Since the procedural problems did 
not already provide the scaffolding for conceptual knowledge acquisition and sense 
making, the students may have found support for sense making through collaboration. 
The results of more joint attention and longer hint duration in the procedural condition 
and not the conceptual condition support this position. 

By triangulating three data streams, we see preliminary evidence of good collabo¬ 
ration within the procedural condition. Currently these measures of collaboration are 
imperfect and more analysis is needed to pinpoint the somewhat surprising trends we 
have seen of better collaboration during procedural tasks. By continuing analysis on 
the transcripts, specifically contrasting cases where we are seeing evidence of this 
good collaboration with those where we do not, patterns may emerge of what success¬ 
ful dyads are doing compared with less successful dyads. This dialogue data along 
with the tutor log data may provide insights into how best to support collaboration 
with elementary school students. It is important to look at both the design features that 



may correspond with moments of good collaboration and to also find patterns in good 
collaborating dyads that are not present in other dyads that could then be better sup¬ 
ported in the problems. Future work will focus on analyzing the dialogue and tutor log 
data to find features to better support collaboration with elementary school students. 
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